Robustness and Conditional Independence Ideals

We study notions of robustness of Markov kernels and probability distribution of a system that is described by $n$ input random variables and one output random variable. Markov kernels can be expanded in a series of potentials that allow to describe the system's behaviour after knockouts. Robustness imposes structural constraints on these potentials. Robustness of probability distributions is defined via conditional independence statements. These statements can be studied algebraically. The corresponding conditional independence ideals are related to binary edge ideals. The set of robust probability distributions lies on an algebraic variety. We compute a Gr\"obner basis of this ideal and study the irreducible decomposition of the variety. These algebraic results allow to parametrize the set of all robust probability distributions.Comment: 16 page

Secret Sharing and Shared Information

Secret sharing is a cryptographic discipline in which the goal is to distribute information about a secret over a set of participants in such a way that only specific authorized combinations of participants together can reconstruct the secret. Thus, secret sharing schemes are systems of variables in which it is very clearly specified which subsets have information about the secret. As such, they provide perfect model systems for information decompositions. However, following this intuition too far leads to an information decomposition with negative partial information terms, which are difficult to interpret. One possible explanation is that the partial information lattice proposed by Williams and Beer is incomplete and has to be extended to incorporate terms corresponding to higher order redundancy. These results put bounds on information decompositions that follow the partial information framework, and they hint at where the partial information lattice needs to be improved.Comment: 9 pages, 1 figure. The material was presented at a Workshop on information decompositions at FIAS, Frankfurt, in 12/2016. The revision includes changes in the definition of combinations of secret sharing schemes. Section 3 and Appendix now discusses in how far existing measures satisfy the proposed properties. The concluding section is considerably revise

Finding the Maximizers of the Information Divergence from an Exponential Family

This paper investigates maximizers of the information divergence from an exponential family $E$. It is shown that the $rI$-projection of a maximizer $P$ to $E$ is a convex combination of $P$ and a probability measure $P_-$ with disjoint support and the same value of the sufficient statistics $A$. This observation can be used to transform the original problem of maximizing $D(\cdot||E)$ over the set of all probability measures into the maximization of a function \Dbar over a convex subset of $\ker A$. The global maximizers of both problems correspond to each other. Furthermore, finding all local maximizers of \Dbar yields all local maximizers of $D(\cdot||E)$. This paper also proposes two algorithms to find the maximizers of \Dbar and applies them to two examples, where the maximizers of $D(\cdot||E)$ were not known before.Comment: 25 page

Generalized Binomial Edge Ideals

This paper studies a class of binomial ideals associated to graphs with finite vertex sets. They generalize the binomial edge ideals, and they arise in the study of conditional independence ideals. A Gr\"obner basis can be computed by studying paths in the graph. Since these Gr\"obner bases are square-free, generalized binomial edge ideals are radical. To find the primary decomposition a combinatorial problem involving the connected components of subgraphs has to be solved. The irreducible components of the solution variety are all rational.Comment: 6 pages. arXiv admin note: substantial text overlap with arXiv:1110.133

The Blackwell relation defines no lattice

Blackwell's theorem shows the equivalence of two preorders on the set of information channels. Here, we restate, and slightly generalize, his result in terms of random variables. Furthermore, we prove that the corresponding partial order is not a lattice; that is, least upper bounds and greatest lower bounds do not exist.Comment: 5 pages, 1 figur

Hierarchical Models as Marginals of Hierarchical Models

We investigate the representation of hierarchical models in terms of marginals of other hierarchical models with smaller interactions. We focus on binary variables and marginals of pairwise interaction models whose hidden variables are conditionally independent given the visible variables. In this case the problem is equivalent to the representation of linear subspaces of polynomials by feedforward neural networks with soft-plus computational units. We show that every hidden variable can freely model multiple interactions among the visible variables, which allows us to generalize and improve previous results. In particular, we show that a restricted Boltzmann machine with less than $[ 2(\log(v)+1) / (v+1) ] 2^v-1$ hidden binary variables can approximate every distribution of $v$ visible binary variables arbitrarily well, compared to $2^{v-1}-1$ from the best previously known result.Comment: 18 pages, 4 figures, 2 tables, WUPES'1